Large-scale network environments typically involve the storage of data in one or more data stores. Present network systems often utilize storage nodes to coordinate input/output (I/O) access to central storage resources by multiple clients. In this manner, multiple and often simultaneous read/write requests from multiple clients can be synchronized and executed by a single device. In cases where a single device (e.g., storage node) must process a number of parallel data streams, it can currently do so using one of several techniques. First, a single I/O (input/output) process in which multiple data streams are multiplexed and a single stream is written. This is typically used for backup to tape or other sequential access devices, and does not fit disk-based use cases. Second, a single I/O process that writes multiple data streams in parallel by allocating time to each stream. This is typically used when multiple data streams are to be written to same target disk volume. The drawback of this method is saturation of a process with low number of streams and decreased performance as the stream count increases. This method also creates a limiting value on how many streams can be processed in parallel. Third, multiple I/O processes which each write a single data stream. This method is typically used when multiple data streams are written to separate disk volume targets. The drawback of this method is that the customer must manage a large number of disk volume targets, which imposes large overhead requirements, as well as imposes limits on the ability to access the data in timely manner. Existing methods of processing multiple parallel read/write requests in the disk storage environment thus utilize either a multi-thread model only or a multi-process model only to process a number of parallel data streams. This limited approach generally does not provide an adequate level of load balancing among multiple processes and limits the amount of concurrency available in most systems, thus reducing the efficiency and performance of these systems.
What is needed therefore is an input/output access method that provides full control on desired I/O access pattern, unlimited concurrency for all types of read or write operations, and increased I/O performance due to process load balancing. Such a method is provided by a system that implements a hybrid multi-thread and multi-process model that combines the advantageous features of each model. Although such a hybrid multi-thread/multi-process approach has been utilized in applications such as processing web requests, this approach has not been used in the network storage environments.